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A  MODEL  SAMPLING  EXPERIMENT  TO  EVALUATE  TWO  METHODS 
OF  TEST  SELECTION 


^  BACKGROUND  AND  PURPOSE 

Assignment  of  Army  enlisted  personnel  to  Advanced  Individual.  Training 
following  basic  training  is  based  on  a  battery  of  tests  called  the  Army 
Classification  Battery  (ACB).  Currently,  the  ACB  scores  are  combined  in 
pairs  to  form  aptitude  area  scores  each  of  which  is  considered  predictive 
of  performance  for  the  several  Military  Occupational  Specialties  (MOS) 
within  one  or  more  occupational  categories.  Optimal  assignment  of  enlisted 
men  to  these  specialties  is  based  on  these  estimates  of  performance. 

As  part  of  the  Army  research  program  to  improve  assignment  of  personnel, 
new  test  batteries  to  complement  or  replace  existing  test  batteries  are 
continually  being  developed.  A  procedure  generally  used  in  the  development 
of  test  batteries  involves  l)  the  formation  of  an  experimental  battery  com¬ 
posed  of  a  large  number  of  tests — the  test  pool — and  2)  the  selection  of 
some  subset  of  the  pool  to  form  an  operational  battery.  Selection  of  tests 
to  predict  a  single  criterion  of  job  performance  is  relatively  simple  and 
straightforward ;  tests  are  usually  sequentially  selected  to  maximize  the 
multiple  correlation  coefficient  under  the  constraint  that  all  previously 
selected  tests  are  retained.  However,  the  selection  of  tests  to  predict 
several  criteria  can  be  accomplished  by  several  different  methods.  The 
present  Research  Memorandum  describes  the  use  of  model  sampling  techniques 
to  evaluate  two  of  these  test  selection  methods. 


TEST  SELECTION  PROCEDURES 

The  two  test  selection  methods  compared  in  this  study  were  developed 
by  Horst:  l)  selection  for  absolute  prediction,  and  2)  selection  for  dif¬ 
ferential  prediction.  The  absolute  method  selects  tests  to  maximize  the 
average  variance  of  the  predicted  criteria  and  is  equivalent  to  maximiz¬ 
ing  the  sum  of  the  squared  multiple  correlation  coefficients  for  predicting 
each  criterion  separately.  The  absolute  method  will  yield  the  battery  having 
the  highest  mean  accuracy  of  prediction  for  all  criteria.  It  is  useful  in 
many  situations,  as  when  personnel  are  selected  if  they  exceed  the  cut-off 
score  on  one  or  more  of  the  performance  estimates  and  their  assignment  is 
determined  from  other  considerations.  The  second  method,  for  differential 
prediction,  selects  tests  to  maximize  the  average  variance  of  the  differences 
between  all  possible  pairs  of  predicted  criteria.  When  criteria  and  pre¬ 
dictors  are  in  standard  form,  this  procedure  is  equivalent  to  maximizing 
the  difference  between  the  average  variance  of  the  predicted  criteria  and 
the  average  of  their  covariances.  Batteries  selected  by  this  method  will 
provide  the  best  prediction  of  relative  success  in  several  jobs. 


STANDARD  QF  COMPARISON 


Whicn  method  is  best  depends  on  the  objective  cf  the  test  selection.  The 
classical  and  most  common  standard  of  comparison  is  tqe  validity  of  tli.  battery 
whoa  a  single  criterion  is  used  (or  the  sum  of  the  squared  validity  coeffi¬ 
cients  when  multiple  criteria  are  used).  Since  the  absolute  method  selects 
tests  specifically  to  maximize  validity,  the  absolute  method  is,  by  definition, 
best  by  this  criteria.  No  other  method  could  possibly  be  superior.  Similarly, 
the  differential  selection  method  would  he  best  if  evaluated  against  the  func¬ 
tion  it  maximizes.  The  function  maximized  by  either  method  would  obviously 
not  provide  a  meaningful  standard  for  comparison  of  the  two  selection  methods. 

The  standard  used  in  the  present  study  was  the  extent  to  which  the  bat¬ 
teries  accomplished  the  purpose  for  which  they  were  constructed- -to  provide 
maximum  improvement  in  the  optimal  assignment  of  personnel  to  Jobs.  The 
batteries  selected  by  the  two  methods  were  therefore  evaluated  on  the  basis 
of  average  expected  performance  resulting  from  optimal  assignment  to  Jobs, 
with  quota  restrictions  on  the  number  assigned  to  each  Job. 


VARIABLES 

The  group  of  tests  comprising  the  test  pool  is  listed  in  Table  1.  The 
order  of  selection  for  the  first  20  tests  by  each  method  is  given  in  paren¬ 
thesis.  The  criteria  used  for  the  test  selection  were  school  final  course 
grades  for  the  12  MOS  listed  in  Table  2. 


PROCEDURES 

This  study  was  carried  out  in  four  steps:  l)  selection  of  tests, 

2)  simulation  of  scores  of  the  selected  tests,  5)  optimal  assignment  based 
on  the  simulated  test  scores,  and  4)  evaluation  ot  the  tests  using  results 
of  optimal  assignment.  Each  of  these  steps  is  discussed  in  turn. 

A  battery  of  32  tests  was  used  as  the  poor  from  which  smaller  batteries 
of  tests  were  selected.  Intercorrelations  calculated  from  a  sample  of  2480 
subjects  were  available,  along  with  coefficients  of  validities  for  12  groups 
of  Jobs,  each  based  on  different  subjects.  Sample  size  in  the  Jobs  ranged 
from  103  to  305-  Absolute  and  differential  test  selection  procedures  were 
applied  to  the  pool,  and  batteries  of  size  5,  10,  and  20  tests  were  selected 
by  each  of  the  two  methods.  These  six  batteries  are  henceforth  referred  to 
as  the  selected  batteries. 
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Table  1 


BASIC  SET  OF  TESTS  AVAILABLE  FOR  SELECTION  BY  TWO  METHODS4 


Army  Classification  Battery 


Verbal  (20,  4)b 
Arithmetic  Reasoning  (l,  11 ) 
Pattern  Analysis 
Mechanical  Aptitude 
Army  Clerical  Speed  (l8,  17 ) 
Army  Radio  Code  (11,  20) 


Shop  Mechanics 

Automotive  Information  (2,  7) 
Electronics  Information  (3,  -) 
Classification  Inventory  ( -,  2) 
General  Information  (12,  9) 


Army  Differential  Aptitude  Series 


Object  Completion  (15,  14) 
Word  Squares 
Pattern  Analysis  (-,  8) 
Practical  Situations 
Reaction  to  Signals 
Mechanical  Principles  (9,  18) 
Spatial  Orientation 


Letter  Combinations 
Hidden  Figures 

Attention  to  Detail  (10,  16) 
Patterns 

Perceptual  Speed  (6,  3) 
Associative  Memory  (l4,  -) 
Subtraction  and  Division  (l6,  10 ) 


Noncognitive 

Clerk  a  priori  (17,  15) 

Electronics  a  priori  (8,  l) 

Mechanic  a  priori  (5>  5) 

Mechanic  Suppressor  (4,  6) 


Scales 

General  Adjustment  Empirical  (19,  19) 
Clerk  Empirical  (7,  12) 

Mechanic  Empirical  (13,  13) 


aThe  test  selection,  done  by  Dr.  William  H.  Relme,  was  reported  in  Technical 
Research  Note  155,  Evaluation  of  Differential  Classification  Tests  for  the 
ACB.  June,  1965*  U.  S.  Army  Personnel  Research  Office,  Washington,  D.  C. 

23315- 

bThe  first  number  in  parenthesis  is  order  of  selection  for  absolute  pre¬ 
diction;  the  2nd  number  is  order  for  differential  prediction. 


Table  2 


CRITERION  VARIABLES  USED  FOR  TEST  SELECTION 


MOS  No. 

Approximate 
equivalent  in 
current  .lob  structure 

Title 

223.1 

Missile  electronics  mechanic 

250.0 

Electronics  repair  helper 

271.1 

32B 

Fixed  station  receiver  repairman 

281.1 

26L 

Microwave  radio  repairman 

293.1 

31M 

Radio  relay  and  carrier  operator 

294.1 

31L 

Field  carrier  equipment  repairman 

296.1 

31E 

Field  radio  repairman 

310.0 

36A 

Field  communication  crewman 

321.1 

36F 

Lineman 

352.1 

Engineer  missile  equipment  speclalis 

440.0 

44A 

Metalworking  helper 

551.1 

533 

Carpenter 

530.0 

54A 

Chemical  operations  helper 

550.0 

56a 

Supply  handler 

626.1 

62E 

Construction  machine  operator 

627.1 

62F 

Crane  shovel  operator 

670.0 

67A 

Aircraft  maintenance  crewman 

680.0 

68a 

Aircraft  components  repair  helper 

723.1 

72B 

Conmuni cations  center  specialist 

724.1 

72c 

Switchboard  operator 

053.1 

05c 

Radio  teletype  operator 

Samples  of  personnel  were  simulated  using  the  assumption  that  the  test 
scores  could  be  represented  by  normal  distributions.  Hence,  by  model* 
sampling  techniques,  scores  based  on  each  of  the  six  batteries  were  generated 
by  computer  to  have  the  statistical  characteristics  of  samples  from  a  normal 
population.  In  addition,  the  covariance  matrices  of  the  normal  populations 
used  to  generate  the  scores  were  equal  to  the  covariance  matrices  of  the  six 
selected  batteries.  Either  a  5-  10-  or  20-variable  normal  distribution  was 
used  for  the  sample,  depending  on  the  size  of  the  simulated  selected  battery. 
Ten  samples  of  216  subjects,  each  sample  having  different  normally  distributed 
scores  were  generated  for  each  selected  battery.  Different  normal  variates 
(scores)  were  used  for  batteries  of  different  size,  but  within  each  battery 
size  the  same  normal  variates  were  used  for  batteries  selected  by  each  method. 
Thus,  the  same  simulated  subjects  were  used  in  absolute  and  differential  bat¬ 
teries  of  the  same  size. 

Assignment  of  simulated  subjects  to  Jobs  was  based  on  estimates  of  per¬ 
formance  for  each  subject  in  each  Job.  The  performance  estimates  were  cal¬ 
culated  separately  for  each  selected  battery  from  the  simulated  test  scores 
by  the  least-squares  regression  equations.  Uniform  quotas  were  used  in  the 
assignment. 

Assignment  was  evaluated  using  least  squares  regression  estimates  of 
performance,  based  on  all  32  tests,  in  the  Job  to  vhich  each  subject  was 
assigned.  These  performance  estimates  were  averaged  over  subjects,  yielding 
the  assignment  average.  Since  10  samples  were  generated  and  assigned  for 
each  selected  battery,  10  assignment  averages  were  calculated  for  each  bat¬ 
tery.  Each  of  the  six  sets  of  10  assignment  averages  was  then  averaged  to 
obtain  six  mean  assignment  averages;  these  mean  assignment  averages  were  used 
to  compare  the  assignment  effectiveness  of  the  six  selected  batteries. 

The  differences  between  the  mean  assignment  averages  for  the  six  selected 
batteries  were  tested  for  significance  by  the  t-test  for  correlated  means. 

The  mean  assignment  averages  and  the  t-test  values  are  shown  in  Table  3* 

Under  random  allocation,  the  mean  assignment  averages  would  have  a  mean  of 
100  and  a  standard  deviation  of  20. 

Table  3 

MEAN  ALLOCATION  AVERAGES* 


Battery  Size 

Absolute 

Differential 

t- value 

5 

109-79 

110.89 

6.45* 

10 

112.99 

113.24 

4.76* 

20 

-  m-99 

114.15 

9.76*  _ 

♦Significant  at  the  .01  level. 


^he  allocation  average  is  the  objective  function  for  the  linear  programming 
model  divided  by  the  size  of  the  sample  on  vhich  objective  function  is 
maximized. 
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RESULTS 


Examination  of  Table  3  shove  that  differential  selection  resulted  In  the 
■ore  efficient  asslgnaent  for  all  three  battery  sites.  This  finding  is 
interesting  in  view  of  the  pattern  of  selected  test:  absolute  and  differ¬ 
ential  batteries  of  3  tests  bad  only  1  test  in  caagaon  (20f>),  while  the  bat¬ 
teries  of  20  tests  had  l8  tests  in  cccsaon  (90^).  The  difference  be  ween 
batteries  of  20  tests  was  more  significant  than  was  the  difference  between 
the  5-test  batteries  In  spite  of  the  much  greater  overlap  between  20-test 
batteries.  Rote  the  relationship  between  assignment  efficiency  and  validity 
(Table  4  and  5)«  The  absolute  battery  of  5  tests  had  substantially  higher 
validity  (for  each  MOS)  than  did  the  differential  battery;  validity  coef¬ 
ficients  for  the  20-teBt  batteries  were  very  similar.  Yet  for  both  bat¬ 
teries  the  differential  battery  resulted  in  more  efficient  assignment. 

In  suanary,  when  a  test  battery  is  to  be  used  for  optimal  assignment 
of  personnel,  the  evidence  presented  shows  clearly  the  superiority  of  dif¬ 
ferential  selection  over  absolute  selection,  at  least  for  the  variables 
vised.  Studies  are  now  being  conducted  to  establish  the  generality  of 
this  finding  using  test  pools  of  several  different  sires  and.  several  dif¬ 
ferent  numbers  of  criteria.  Future  studies  will  also  use  criteria  covering 
another  range  of  content,  in  an  effort  to  establish  the  generality  of  the 
finding. 
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battery  of  15  tests  was  not  simulated;  Validity  coefficients  for  a  15-test  battery  are  included  as 
incidental  information. 
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a0p*  cit.,  see  Table  2, 

^Decimal  points  omitted. 
cM0S  titles  given  in  Table  2. 

dA  battery  of  15  tests  was  not  simulated;  Validity  coefficients  for  a  15- test  battery  are  included 
as  incidental  information. 


